Effective Interventions Unit 
Evaluation Guide 9 
Evaluating employability programmes 



WHAT IS THE PURPOSE OF THIS GUIDE? This is the ninth evaluation guide in the El U evaluation 
series. It explains the aims and objectives of employability projects and associated evaluations. It 
focuses on "soft" employability outcomes, indicators and tools for measuring them. 

WHO SHOULD READ IT? Anyone involved in commissioning, planning, developing, delivering and 
evaluating services which aim to support recovering drug users into employment, training and 
education. 

THE El U WOULD LI KE TO THANK HARI S MARTI NOS AND HELEN J UDGE AT LOCAL AND REGI ONAL 
DEVELOPMENT PLANNI NG (LRDP) FOR THEI R HELP AND EXPERTI SE I N PREPARI NG THI S GUI DE. 



WHAT I S AN EMPLOYABI LITY PROGRAMME? 

Employability programmes are projects or interventions to help drug users and other disadvantaged groups to 
become more "employable" and to progress towards the labour market and ultimately a job. As the EIU 
review "Moving On" has shown, the vast majority of the clients are unlikely to be ready for open market 
employment at the end of active in-programme support. Therefore, employability programmes aim to support 
clients to progress along a pathway to employment. 

There is a wide range of programmes offering support towards employability. They provide counselling, training 
and other one-to-one and group activities mainly aimed at personal and basic skills development. Referrals and 
after care support are important aspects. In the wide spectrum of services and projects supporting drug users and 
recovering drug users, specialist employability projects occupy a place between treatment and rehabilitation 
services and mainstream education, training and employment services, as illustrated below. There are 
however many links and overlaps between employability and other types of support. There are stand alone 
employability projects as well as cases where employability support is combined with other services, representing 
an extension of treatment/ rehabilitation or an add-on to formal training or education. 



Types of support services and projects for drug users 


Treatment and 
Rehabilitation Services 


Employability Programmes 


Mainstream Education, 
Training and Employment 
Services 


• Community rehabilitation services 

• Social Work Services 

• Harm reduction services 

• Community Pharmacy 

• Residential rehabilitation 

• Prison based services 


• New Futures Fund projects 

• SIP projects 

• Progress & Progress2work 

• Beattie 1 nclusiveness Pilots 

• Supported employment 

• Intermediate labour market 

• Post- release programmes for 
prisoners 


• New Deal 

• Enterprise Network Training 
Programmes 

• Further & Higher Education 
1 nstitutions 

• Community Education 



WHAT ARE THE Al MS OF EVALUATI ONS OF EMPLOYABI LI TY PROGRAMMES? 

The main aim of the evaluation of an employability programme or project is to establish whether it has achieved its 
objectives, i.e. whether (and how far) its clients have progressed towards greater employability and ultimately 
employment. 

To be able to establish progress made in terms of employability, the programme and the evaluation exercise must 
have a clear concept of "pathways to employment" for the client. A key aspect of the evaluation of 
employability projects is the measurement of "distance travelled" , i.e. the progress that the clients have made in 
terms of greater employability on their pathway towards the labour market and employment, as a result of the 
intervention. I n order to measure "distance travelled" we need to establish the types of outcomes that clients are 
expected to achieve and to define indicators suitable for measuring these outcomes. 
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"HARD" OUTCOMES AND INDICATORS 

Some clients may achieve hard outcomes, such as moving into training or further education or even getting a 
job. In this case, the use of indicators is relatively straightforward - the numbers and proportions of people 
moving into education or getting a job are typical hard indicators. 



"SOFT" OUTCOMES AND I NDI CATORS 

Many more clients are likely to achieve soft outcomes, such as improved motivation/confidence, personal or work 
skills. Soft outcomes tend to be subjective and a matter of degree rather than absolute, and therefore much more 
difficult to measure. However, gradually a number of soft indicators (both qualitative and quantitative) have 
been used in employability projects. Although unlikely to measure with precision the progress made, they can 
provide firm indications regarding the achievement of, and progress towards, an outcome. For example, improved 
levels of attendance and improved communication skills can suggest strongly that motivation has increased. 



There is no standardisation or generally used typology of outcomes and indicators. The following summarise the 
different types of soft outcomes that are found in employability projects which support a variety of client groups 
and can be treated as "core" outcomes. 



Examples of core outcomes and indicators 


Types of core outcomes 


Examples of indicators 


Personal Development 
Social Competence 


• Confidence, self esteem 

• Motivation, increased feelings of responsibility 

• Higher personal and career aspirations 

• Relationships with peers and authority 

• (Basic) interpersonal and communication skills 

• Team working 


Basic Work Skills and Attributes 


• Basic literacy (reading, writing) 

• Basic numerical skills, including ability to manage money 

• Timekeeping, reliability 

• Ability to complete forms, CV writing 

• Presentation 


Core Skills 


• Communication 

• Numeracy 
ICT 

• Problem solving 

• Interpersonal 


Personal Effectiveness and Aptitude 


• Planning 

• Prioritising 

• Verbal reasoning 

• Numerical reasoning 



Intermediate steps (milestones) should be defined for achieving certain levels of outcome. Attainment of such 
levels should allow the client to move on to a new stage of support within the project, targeting a higher level of 
employability outcomes, or to his or her next destination outside the project. For example, people who made 
substantial progress in terms of personal development and basic skills should be able to progress to a higher level 
of skills and aptitudes (represented by Core Skills etc) and become "labour market ready" or "New Deal ready". 

Some of the soft outcomes are particularly relevant or apply specifically to certain client groups. In the case of 
clients with substance misuse problems there are some group- specific outcomes, which are desirable and will 
assist the individual in moving on in terms of employability. These are likely to include: 

Gradual reduction in drug use 
Improved physical and psychological health 
Improved personal and family relationships 
Reduction in impact of negative peer pressure 
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TOOLS FOR MEASURI NG OUTCOMES AND Dl STANCED TRAVELLED 

In order to measure (or otherwise assess) an outcome and the overall distance travelled by clients towards the 
labour market and employment, we need to: 

• define appropriate indicators and outcomes; and 

• obtain and analyse data measuring the value of these indicators at different stages of the intervention. 

As stated earlier, in the case of hard outcomes and indicators it is easy to do so by using quantitative 
information on jobs, qualifications, and training / further education places. A major concern among staff about 
soft outcomes and indicators is that many are difficult, if not impossible, to measure, as they are often not 
quantifiable. Soft outcomes are largely intangible and they cannot be measured with the same ease as numbers of 
jobs and qualifications. 

Tools for measuring soft outcomes, therefore, are not standardised and vary considerably from project to 
project. There is a substantial number of tools which can be used by employability programmes or projects either 
as already developed or after adaptation to suit the particular requirements of projects. For example, the Rickter 
Scale has been used by projects supported by the New Futures Fund (see EIU Evaluation Guide 7). 

Such tools have been presented systematically in a digest prepared for the Scottish Executive by the University of 
Strathclyde (see Further Resources, below) and some examples are summarised in the table below. Many of these 
tools are "assessment" tools whilst others cover both "assessment" and "development". Some are comprehensive 
frameworks whilst others cover only particular aspects of employability. Most of them have features that can be 
used, with or without further adaptation, for the evaluation of employability outcomes. 



Examples of tools with features suitable for measuring emplovabilitv outcomes 


Type of 
outcome 


Tool 


Features 


Basic Skills 


Wordpower & Numberpower 

Further information: 
enauirvOcitv-and- auilds.co.uk 


Two complementary tools to assess ability level, and 
enhance basic skills development, and measure progress 
in basic literacy and basic numerical skills (every day 
application of numbers including money and time). Three 
different levels are available. 


BeST (2) 

Further information: 
be-consultrabtinternet.com 


Screening toolkit for Assessing basic skills, i.e. reading, 
writing, speaking English, numbers. Designed to test 
people using the Basic Skills Standards as exemplified in 
Wordpower and Numberpower. (Interactive multi media 
CD ROM). 


Personal 
Development 
and Social 
Competence 


The Rickter Scale (3) 

Further information: 

ricktercabla kestevenson . demon . co . u k 


A non-paper based assessment and evaluation tool (a 
colourful plastic board) that allows clients to explore their 
circumstances, identify priority areas for support and 
assess progress across time. Aspects covered include: 

• Life Board 

• Personal Social Development 

• Drug/ Alcohol Issues 

• Preparation for Work 


Social Skills Training 

Further information: 
edu&hscranfer- nelson .ro. uk 


Assesses social competence and helps develop 
intervention programmes to raise levels of interpersonal 
skills for young people. 


Mult i- faceted 
Frameworks 


Bridges to Progress 

Further information: 
bridaesproj ectOcompuserve.com 


A model for measuring individual progress towards self- 
reliance and employability, focused on soft measurement 
and on progress, not deficit. Aspects covered include: 

• Skills for self-reliance 

• 1 ndicators of vulnerability 


The Connexions Framework 

Further information: 
dfes®proloqistics. co.uk 


A framework for coordinating a range of assessment 
information on a client, covering a comprehensive range 
of factors, including: 

• Life, Basic and Key skills 

• Emotional well-being; Physical health 

• Income; Housing 

• Identity and self-image; Attitudes and motivation 



For information on other tools see the Digest, under Further Resources at the end of this Guide 
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PROCESS EVALUATI ON 



It is often not sufficient to evaluate the outcomes of a programme. Process evaluation is also important. It 

seeks to find out how an intervention works and why something worked well. The definition of 'process evaluation' 
and techniques commonly used for evaluating these aspects are discussed in EIU Evaluation Guides 1 and 2. This 
part of the evaluation should cover aspects such as: 

Publicity, promotion, client recruitment (did we get the right clients?) 

Project activities and method of support, including client-centred action plans (did we address their main needs 
and did we help them acquire/develop skills and aptitudes suitable for moving on towards employment?) 
Project management and resources (did the project have sufficient and appropriate financial and human 
resources and systems?) 

USING FINDINGS / DRAWING LESSONS 

In general terms, the evaluation of a programme should be able to establish the extent to which its objectives 
have been achieved (and the reasons that contributed to success or otherwise). Once the outcomes of the 
employability project have been identified, the evaluation should attempt to interpret these findings, by looking at 
the process evaluation and other contextual factors. Key issues to consider in interpreting findings, include: 

• To arrive at project- wide (and programme- wide) outcome measures it is essential to aggregate (present 
data as totals) and analyse information collected on the progress achieved / distance travelled by individual 
clients. In the case of soft indicators, projects should define common milestones for all their clients along 
their pathway to employment. 

• The sustainability of the outcomes should be taken into account in drawing conclusions. Evidence of 
sustainability could take the form of clients staying in education or volunteering, or moving on to harder 
outcomes, such as obtaining qualifications or getting a job. 

It is important to look beyond the project or intervention itself, to form a view as to the extent to which the 
outcomes are a direct result of the intervention. The following could be useful (1) make a comparative 

analysis of its outcomes with those of other similar interventions (2) compare its outcomes with a control 
group of people similar to the clients of the project, who did not receive employability support; (3) take 
account of contextual factors, such as changes in the local employment situation and the skills and aptitudes 
required by employers. 



SUMMARY 

• Evaluations of employability programmes or projects examine the extent to which their clients have progressed 
("distance travelled") towards greater employability, and ultimately employment. 

• There are core outcomes and indicators which are commonly used in evaluating employability projects. Many of 
these are "soft" outcomes, such as improved motivation/confidence. 

• There are a number of tools that can be used, with or without adaptation, for measuring soft outcomes. 



FURTHER RESOURCES 

• Centre for Guidance, Careers and Personal and Social Development, University of Strathclyde (2001), 
Identifying Learning and Support Needs: a digest of assessment tools, prepared for the Scottish 
Executive (Beattie: Implementing I nclusiveness ) 

• Dewson S, Eccles J , Tackey N D, J ackson A (2000) Measuring Soft Outcomes and Distance Travelled: A 
Review of Current Practice, Research Report RR219 DfEE 

• Effective Interventions Unit, Scottish Executive (2001) Moving On: Education, training and employment 
for recovering drug users 



